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DETAILED ACTION 

Continued Examination Under 37 CFR 1.114 

1 . A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1 .17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02 
February 2010 has been entered. 

Response to Arguments 

2. Applicant's arguments filed 25 January 2010 have been fully considered but they 
are not persuasive. 

In response to applicant's arguments that the given references do not teach 
"divides extracted... identifies flesh color", Page 14, lines 4-9, the examiner 
respectfully disagrees. Kim teaches extracting a skin color image from a still 
image video frame and dividing the image into a grid of blocks (Col. 4, lines 35- 
56). Tow teaches dividing an image into blocks and calculating a motion vector 
for each of the blocks (Col. 8, lines 14-65). Okada teaches dividing the image 
into blocks, extracting a flesh color image from a still image video frame, and 
calculating a motion vector for each of the blocks (Col. 7, lines 16-40; Col. 8, 
lines 15-59; Col. 9, lines 9-14, 40-52). The only pixels that are extracted from the 
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original image are pixels having a flesh-color. Therefore, there are no pixels 
other than the flesh-colored ones in the extracted image. 



Claim Rejections - 35 USC § 101 
3. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

Claims 57-58 are rejected under 35 U.S.C. 101 because the claimed invention is 
directed to non-statutory subject matter. Claim 58 sets forth a "computer readable 
media." However, the specification as originally filed and applicant's response received 
03 March 2009 (Section II. Rejections under 35 U.S.C. 101 - Pages 11 -12) does not 
explicitly define the computer readable media. The United States Patent and Trademark 
Office (USPTO) is obliged to give claims their broadest reasonable interpretation 
consistent with the specification during proceedings before the USPTO. See In re Zletz, 
893 F.2d 319 (Fed. Cir. 1989) (during patent examination the pending claims must be 
interpreted as broadly as their terms reasonably allow). The broadest reasonable 
interpretation of a claim drawn to a computer readable media (also called machine 
readable medium and other such variations) typically covers forms of non-transitory 
tangible media and transitory propagating signals per se in view of the ordinary and 
customary meaning of computer readable media, particularly when the specification is 
absent an explicit definition or is silent. See MPEP 21 1 1 .01 . When the broadest 
reasonable interpretation of a claim covers a signal perse, the claim must be rejected 
under 35 U.S.C. § 101 as covering non-statutory subject matter. See In re Nuijten, 500 
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F.3d 1346, 1356-57 (Fed. Cir. 2007) (transitory embodiments are not directed to 
statutory subject matter) and Interim Examination Instructions for Evaluating Subject 
Matter Eligibility Under 35 U.S.C. § 101, Aug. 24, 2009; p. 2. 

Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

5. Claims 1,2,4, 5, 29-34, 55-58 are rejected under 35 U.S.C. 1 03(a) as being 
unpatentable over Kim (US 6,289,110) in view of Tow (US 7,266,771) and further in 
view of Okada (US 5,907,361 ). 

Regarding claim 1, Kim teaches an audience state estimation system (Fig. 
1) comprising: 

imaging device for imaging an audience and generating a video signal 
relative to the audience thus imaged (Col. 2, line 61 -Col. 3, line 4; Col. 5, lines 
36-47); 

movement amount detection device for detecting a movement amount of 
said audience based on said video signal, i.e. determining motion information 
from video frames (Col. 3, lines 1-4, 14-29; Col. 4, lines 26-35); 
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wherein the movement amount detection device extracts a flesh-color 
area which identifies flesh color from said video signal, i.e. generating a skin 
color image frame from the video (Col. 3, lines 5-30; Col. 4, lines 35-55), 

divides the extracted flesh-color area into blocks identifying flesh color 
(Col. 3, lines 14-45; Col. 4, lines 50-65; Col. 5, lines 17-30), and 

estimation device for estimating an audience state based on said 
movement amount, i.e. determining the nearest person looking at the camera 
(Col. 5, lines 17-30, 36-47). 

Kim does not clearly teach the movement amount detection device 
calculates movement vectors for each of the divided blocks identifying flesh 
color, wherein each pixel in the divided blocks identifies flesh color; and an 
estimation device for estimating an audience state based on a comparison result 
of said movement amount and a predetermined reference level. 

Tow teaches dividing an area into blocks (Fig. 2, el. 201, 203, 205, 207, 
209; Col. 8, lines 14-65), 

calculating movement vectors for each of the blocks (Fig. 2, el. 201 , 203, 
205, 207, 209; Col. 8, lines 14-65); and 

an estimation device for estimating an audience state based on a 
comparison result of a movement amount and a predetermined reference level, 
i.e. using a motion information template that corresponds to clapping (Col. 10, 
line 46-Col. 11, line 3). 
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Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Kim to include dividing an area into 
blocks, calculating movement vectors for each of the blocks; and an estimation 
device for estimating an audience state based on a comparison result of said 
movement amount and a predetermined reference level, as taught by Tow, for 
the purpose of identifying a type of motion energy at a particular place in a video 
and presenting it to a viewer in a graphically intuitive manner (Tow-Col. 3, lines 
29-35). 

Kim in view of Tow does not clearly teach wherein each pixel in the 
divided blocks identifies flesh color. 

Okada teaches a movement amount detection device extracts a flesh- 
color area which identifies flesh color from said video signal, i.e. an area is 
extracted based on the color of a face (Col. 7, lines 16-40; Col. 8, lines 15-27), 

divides the extracted flesh-color area into blocks identifying flesh color 
(Col. 7, lines 44-67; Col. 8, lines 33-55), and 

calculates movement vectors for each of the divided blocks identifying 
flesh color (Col. 8, lines 55-59; Col. 9, lines 9-14, 40-52), 

wherein each pixel in the divided blocks identifies flesh color (Col. 7, lines 
27-41; Col. 8, lines 15-55). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Kim in view of Tow's movement 
amount detection device to include the movement amount detection device 



Application/Control Number: 10/602,779 Page 7 

Art Unit: 2427 

extracts a flesh-color area which identifies flesh color from said video signal, 
divides the flesh-color area into blocks identifying flesh color, and calculates 
movement vectors for each of the blocks identifying flesh color, using the known 
method of extracting an area of a frame based on the color of a face, as taught 
by Okada, in combination with the audience estimation system of Kim in view of 
Tow for the purpose of providing improved correlation between an extracted area 
of an image and a preceding image (Okada-Col. 3, lines 23-30). 

Regarding claim 2, Kim in view of Tow in view of Okada teaches the 
movement amount detection device determines movement information of the 
imaged audience based on said video signal (Kim-Col. 3, lines 1-4, 14-29; Col. 4, 
lines 26-35), and 

using MPEG differential frames that store motion information in the form of 
motion vectors obtained by determining the difference between adjacent frames 
(Tow-Col. 6, lines 47-60; Col. 8, lines 14-65); and 

wherein an average movement amount showing an average of 
magnitudes of the movement vectors is set as the movement amount (Tow-Col. 
9, line 55-Col. 10, line 33). 



Regarding claim 4, Kim in view of Tow in view of Okada wherein said 
movement amount detection device determines movement vectors of the imaged 
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audience based on said video signal (Kim-Col. 2, line 62-Col. 3, line 4; Tow-Col. 
6, lines 47-60; Col. 8, lines 14-65); and 

calculating an average movement amount showing an average of 
magnitudes of the movement vectors (Tow-Col. 9, line 55-Col. 10, line 33), and 

wherein a time macro movement amount is set as the movement amount 
of said audience, said time macro movement amount being an average of the 
average movement amounts in a time direction thereof, i.e. the motion vectors 
have a magnitude and direction over the time period of a frame or several frames 
(Tow-Col. 9, line 55-Col. 10, line 33). 

Regarding claim 5, Kim in view of Tow in view of Okada teaches when 
said movement amount is larger than a predetermined level, said estimation 
device estimates said audience state to be in any one of states of beating time 
with the hands and of clapping, i.e. using a motion information template that 
corresponds to clapping (Tow-Col. 10, line 46-Col. 1 1 , line 3). 

Regarding claims 29, 30, 55-58, claims are analyzed with respect to claim 
1 . These are Markush claims that include estimating the audience state based 
on audio taken from the audience, data of which was assigned to the non-elected 
Group II of the Restriction/Election Requirement, and likewise all limitations 
dealing with audio will not be examined. 
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Regarding claim 31 , claim is analyzed with respect to claim 1 . 

Regarding claim 32, claim is analyzed with respect to claim 2. 

Regarding claim 33, claim is analyzed with respect to claim 4. 

Regarding claim 34, claim is analyzed with respect to claim 5. 

6. Claims 6-8 and 35-36 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Kim in view of Tow in view of Tagawa (US 7,373,209) and further in view of Okada. 

Regarding claim 6, Kim teaches an audience state estimation system (Fig. 
1) comprising: 

imaging device for imaging an audience and generating a video signal 
relative to the audience thus imaged (Col. 2, line 61 -Col. 3, line 4; Col. 5, lines 
36-47); 

movement detection device for detecting movement of said audience 
based on said video signal (Col. 3, lines 1-4, 14-29; Col. 4, lines 26-35); 

wherein the movement detection device extracts a flesh-color area which 
identifies flesh color from said video signal (Col. 3, lines 5-30; Col. 4, lines 35- 
55), 

divides the extracted flesh-color area into blocks identifying flesh color 
(Col. 3, lines 14-45; Col. 4, lines 50-65; Col. 5, lines 17-30); and 
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estimation device for estimating an audience state based on the 
movement of said audience (Col. 5, lines 17-30, 36-47). 

Kim does not clearly teach a movement periodicity detection device for 
detecting movement periodicity of said audience based on said video signal; the 
movement periodicity detection device extracts a flesh-color area which identifies 
flesh color from said video signal, divides the flesh-color area into blocks 
identifying flesh color, and calculates movement vectors for each of the blocks 
identifying flesh color; and estimation device for estimating an audience state 
based on the movement periodicity of said audience, wherein each pixel in the 
divided blocks identifies flesh color. 

Tow teaches using MPEG differential frames that store motion information 
in the form of motion vectors obtained by determining the difference between 
adjacent frames (Col. 6, lines 47-60; Col. 8, lines 14-65); 

the motion vectors have a magnitude and direction over the time period of 
a frame or several frames, i.e. movement periodicity (Col. 9, line 55-Col. 10, line 
33); 

dividing an area into blocks (Fig. 2, el. 201, 203, 205, 207, 209; Col. 8, 
lines 14-65), 

calculating movement vectors for each of the blocks (Fig. 2, el. 201 , 203, 
205, 207, 209; Col. 8, lines 14-65); and 

estimation device for estimating an audience state based on a comparison 
result of the movement periodicity of said audience and a predetermined 
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reference level, i.e. using a motion information template that corresponds to 
clapping (Col. 10, line 46-Col. 11, line 3). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Kim to include dividing an area into 
blocks, calculating movement vectors for each of the blocks; and wherein an 
average movement amount showing an average of magnitudes of the movement 
vectors is set as the movement amount of said audience, as taught by Tow, for 
the purpose of identifying a type of motion energy at a particular place in a video 
and presenting it to a viewer in a graphically intuitive manner (Tow-Col. 3, lines 
29-35). 

Kim in view of Tow does not clearly teach wherein each pixel in the 
divided blocks identifies flesh color. 

Tagawa teaches detecting periodicity based on an audio signal, i.e. 
identifying a periodicity of a rhythm or beat in music based on the peaks of an 
auto-correlation function of the audio (Col. 13, lines 5-49). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Kim in view of Tow to include a 
movement periodicity detection device for detecting movement periodicity of said 
audience based on said video signal, using the technique taught by Tagawa in 
combination with the motion vector system taught by Kim in view of Tow, 
although in different fields of endeavor would provide a predictable variation to 
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the motion vector system and for the purpose of specifically identifying a type of 
motion for use in a video retrieval system. 

Kim in view of Tow in view of Tagawa does not clearly teach the 
movement periodicity detection device extracts a flesh-color area which identifies 
flesh color from said video signal, divides the flesh-color area into blocks 
identifying flesh color, and calculates movement vectors for each of the blocks 
identifying flesh color. 

Okada teaches the movement amount detection device extracts a flesh- 
color area which identifies flesh color from said video signal, i.e. an area is 
extracted based on the color of a face (Col. 7, lines 16-40; Col. 8, lines 15-27), 

divides the flesh-color area into blocks identifying flesh color (Col. 7, lines 
44-67; Col. 8, lines 33-55), and 

calculates movement vectors for each of the blocks identifying flesh color 
(Col. 8, lines 55-59; Col. 9, lines 9-14, 40-52). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Kim in view of Tow in view of 
Tagawa's movement periodicity detection device to include extracting a flesh- 
color area which identifies flesh color from said video signal, dividing the flesh- 
color area into blocks identifying flesh color, and calculating movement vectors 
for each of the blocks identifying flesh color, using the known method of 
extracting an area of a frame based on the color of a face, as taught by Okada, in 
combination with the audience estimation system of Kim in view of Tow in view of 
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Tagawa for the purpose of providing improved correlation between an extracted 
area of an image and a preceding image (Okada-Col. 3, lines 23-30). 

Regarding claim 7, Kim in view of Tow in view of Tagawa in view of Okada 
teaches the movement periodicity detection device determines movement 
vectors of the imaged audience based on said video signal (Tow-Col. 6, lines 47- 
60; Col. 8, lines 14-65; Tagawa-Col. 13, lines 5-49), 

calculates an average movement amount showing an average of 
magnitudes of the movement vectors (Tow-Col. 9, line 55-Col. 10, line 33), and 

detects an autocorrelation maximum position of the average movement 
amount (Tow-Col. 10, line 57-Col. 11, line 3; Tagawa-Col. 13, lines 5-49), and 

wherein variance of the autocorrelation maximum position is set as said 
movement periodicity (Tagawa-Col. 13, lines 5-49). 

Regarding claim 8, Kim in view of Tow in view of Tagawa in view of Okada 
teaches the variance is calculated using a signal in a frame range, said frame 
range being decided on the basis of the periodicity of said audience state to be 
estimated (Tow-Col. 9, lines 37-55; Col. 10, lines 45-67; Tagawa-Col. 13, lines 5- 
49). 

Regarding claim 35, claim is analyzed with respect to claim 6. 
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Regarding claim 36, claim is analyzed with respect to claim 7. 

7. Claims 9, 10, and 37 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Kim in view of Tow in view of Tagawa in view of Okada and further in view of Lu 
(US 5,550,928). 

Regarding claim 9, Kim in view of Tow in view of Tagawa in view of Okada 
teaches all elements of claim 6. 

Kim in view of Tow in view of Tagawa in view of Okada further teaches a 
movement periodicity (Tow-Col. 9, line 55-Col. 10, line 33; Tagawa-Col. 13, lines 
5-49). 

Kim in view of Tow in view of Tagawa in view of Okada does not clearly 
teach a ratio of low-frequency component in the average movement amount is 
set as said movement periodicity. 

Lu teaches subjecting an image to low-pass filtering (Col. 1 1 , line 48-Col. 
12, line 13). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Kim in view of Tow in view of Tagawa 
in view of Okada to include a ratio of low-frequency component in the average 
movement amount is set as said movement periodicity, using the low-pass 
filtering technique of Lu in combination with the movement periodicity 
determining technique of Kim in view of Tow in view of Tagawa in view of Okada 
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for the purpose of removing extraneous image noise thereby providing a more 
accurate movement periodicity (Lu-Col. 11, lines 63-67). 

Regarding claim 10, Kim in view of Tow in view of Tagawa in view of 
Okada in view of Lu teaches a frequency range of the low-frequency component 
is decided according to the periodicity of the said average movement amount 
transformed to a frequency region to be detected, i.e. identifying the rate and 
rhythm of clapping (Tow-Col. 9, line 55-Col. 10, line 33; Tagawa-Col. 13, lines 5- 
49; Lu-Col. 11, line 48-Col. 12, line 13). 

Regarding claim 37, claim is analyzed with respect to claim 9. 

8. Claims 11 and 38 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Kim in view of Tow in view of Tagawa in view of Okada and further in view of 
Stevens (WO 91/03912). 

Regarding claim 1 1 , Kim in view of Tow in view of Tagawa in view of 
Okada teaches all elements of claim 6. 

Kim in view of Tow in view of Tagawa in view of Okada teaches 
determining the periodicity, rate, and rhythm of a set of motion vectors (Tow-Col. 
10, line 46-Col. 11, line 3; Tagawa-Col. 13, lines 5-49). 

Kim in view of Tow in view of Tagawa in view of Okada does not clearly 
teach the estimation device estimates said audience state to be in a state of 
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beating time with the hands when said movement periodicity is larger than a 
predetermined level, and estimates said audience state to be in a state of 
clapping when said movement periodicity is not larger than said predetermined 
level. 

Stevens teaches estimating a person to be in a state of beating time with 
the hands when said movement periodicity is larger than a predetermined level, 
and estimates a person to be in a state of clapping when said movement 
periodicity is not larger than said predetermined level, i.e. when a first loud or 
sharp sound is received a counter is started; when the counter reaches a 
predetermined level without the system receiving another loud or sharp sound, 
the system determines that the periodicity of the first and a future second sound 
would be too large to be clapping; when a second loud or sharp sound is 
received before the counter reaches the predetermined level, the system 
determines that the periodicity of the two sounds is within a sufficient range to be 
clapping (Abstract). 

Therefore, it would have been obvious to one of ordinary skill in the art at 
the time the invention was made to modify Kim in view of Tow in view of Tagawa 
in view of Okada to include the estimation device estimates said audience state 
to be in a state of beating time with the hands when said movement periodicity is 
larger than a predetermined level, and estimates said audience state to be in a 
state of clapping when said movement periodicity is not larger than said 
predetermined level, using the technique taught by Stevens in combination with 
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the motion vector system taught by Kim in view of Tow in view of Tagawa in view 
of Okada, although in different fields of endeavor would provide a predictable 
variation to the motion vector system and for the purpose of specifically 
identifying a type of motion for use in a video retrieval system. 

Regarding claim 38, claim is analyzed with respect to claim 1 1 . 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to JEREMY DUFFIELD whose telephone number is 
(571)270-1643. The examiner can normally be reached on Mon.-Fri. 8:00 A.M. -5:30 
P.M. EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Scott Beliveau can be reached on (571) 272-7343. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



11 February 2010 
JSD 

/Scott Beliveau/ 

Supervisory Patent Examiner, Art Unit 2427 



